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The  standard  method  of  comparing  k  experimental  treatment  means,  X^3  i  =  1,  fe,  is  to 

construct  an  analysis  of  variance  table  and  conduct  an  F  test.     If  the  F  test  is  significant,  the 
experimenter  can  only  state  that  all  the  means  are  not  equal.     But  he  usually  wants  to  ask  more 
specific  questions  about  differences  among  treatment  means  than  can  be  answered  by  this  F  test . 
In  fact,   the  experimenter  often  wishes  to  decide  which  of  the  true  treatments  means,   u£  =  E(X^) , 
differ  from  each  other. 

Many  methods  are  proposed  in  statistical  literature  for  comparing  experimental  treatment  means 
(Federer  1955;  Hartley  1955;   Scheffe  1959). 2     This  paper  is  a  review  of  that  literature  and  an 
attempt  at  a  unified  presentation  of  multiple  comparison  methods  for  workers  in  the  field. 

The  plan  of  this  paper  is  to  first  provide  a  foundation  on  which  different  multiple  comparison 
methods  may  be  presented,   and  then  to  describe  and  compare  some  well-known  procedures.     The  first 
section,   therefore,  discusses  the  types  of  a-error  rates   (levels  of  significance)  provided  by 
various  multiple  comparison  tests,   and  defines  terms  that  are  used  in  subsequent  sections.  A 
general  multiple  comparison  procedure  is  given  in  section  2.     In  section  3,   five  multiple  compar- 
ison tests  are  individually  described,  and  hints  are  given  for  selecting  the  appropriate  test  for 
a  particular  problem.     Section  4  contains  an  example  problem  illustrating  all  five  methods.  The 
fifth  and  final  section  defines  two  methods  of  testing  multiple  contrasts  among  treatment  means, 
discusses  their  applicability,   and  analyzes  example  data  using  both  procedures.     For  most  forestry 
problems,   the  field  worker  will  find  that  sections   1.3,   1.5,  2,   3.3,   3.4,   3.6,  4.3,  and  4.4  are  of 
the  greatest  value. 


1 .     a-ERROR  RATES  AND  POWER 

The  experimenter  must  attach  some  nominal  level  of  significance  to  his  statements  concerning 
differences  between  treatment  means.     If  he  has  only  two  means  to  compare,   the  usual  "t-test"  at 
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the  a  level  of  significance  will  yield  a  100a  percent  chance  of  stating  that  the  true  means  are 
different  when  they  are  actually  equal.     But  if  the  experimenter  has  k  experimental  means  to  com- 
pare pairwise,     he  is  simultaneously  testing    k^2=  k(k-l)/2  null  hypotheses  of  the  form  H0:\i^  =  uj 
against  the  alternate  hypotheses  E      u^.  4-  Uj  (e.g.,  if  k  =  4,   the  experimenter  may  want  to  make 
six  inferences  at  the  same  time  about  the  values  of        -  u^,         -  u2 ,        -  u3,  u3  ~  M  j ,  v>3  -  u2 , 
and  y2  -  Uj).     An  experiment  yielding  k  means  thus  may  require  as  many  as  fc(fe-l)/2  simultaneous 
"inferences."    The  a  error   (level  of  significance)   for  this  case  has  been  given  more  than  one 
meaning  (Hartley  1955)  .     The  following  definitions  of  error  will  be  encountered  in  this  paper. 

1.1  Error  Rate  Per  Comparison 

ac  =  error  rate  per  comparison  =   (no.  of  erroneous  inferences)   /   (no.  of  inferences  attempted) 
=  proportion  of  all  comparisons  expected  to  be  erroneous  when  the  null  hypotheses  are  true.  For 
example,  with  100  experiments,  each  yielding  20  treatment  means  and  190  paired  comparisons,  the 
experimenter  expects  to  wrongly  reject  950  of  the  19,000  null  hypotheses  if  he  uses  an  ac  error 
rate  of  .05. 

This  error  rate  is  appropriate  when: 

a.  The  specific  comparison  is  the  conceptual  unit. 

b.  The  proportion  of  all  erroneous  inferences   to  all  comparisons  made  is  to  be  a  constant. 

c.  The  experimental  error  variance  is  relatively  stable  from  experiment  to  experiment  (see 
section  1.5). 

d.  A  faulty  inference  does  not  affect  the  remaining  inferences  from  the  experiment  (see 
section  1.5). 

1.2  Error  Rate  Per  Experiment 

ae  =  error  rate  per  experiment  =   (no.  of  erroneous  inferences)   /   (no.  of  experiments)  =  the 
expected  number  of  erroneous  inferences  per  experiment  when  the  null  hypotheses  are  true.  For 
example,  with  100  experiments,  each  yielding  20  treatment  means  and  190  paired  comparisons,  the 
experimenter  expects  to  wrongly  reject  9  or  10  of  the  190  null  hypotheses  in  each  of  the  100 
experiments  if  he  uses  an  ag  error  rate  of  .05. 

This  error  rate  is  appropriate  when: 

a.  The  experiment  is   the  conceptual  unit. 

b.  The  average  number  of  erroneous  inferences  per  experiment  is   to  be  kept  constant. 

c.  The  experimental  error  variance  is  relatively  stable  from  experiment  to  experiment  (see 
section  1.5). 

d.  A  faulty  inference  for  one  comparison  does  not  affect  the  remaining  comparisons  (see 
section  1.5). 

1.3  Experimentwise1*  Error  Rate 

=  experimentwise  error  rate  =   (no.  of  experiments  with  one  or  more  erroneous  inferences)  / 
(no.  of  experiments)  =  expected  proportion  of  experiments  with  one  or  more  erroneous  inferences 
when  the  null  hypotheses  are  true.     For  example,  with  100  experiments,  each  yielding  20  treatment 
means  and  190  paired  comparisons,   the  experimenter  expects  to  wrongly  reject  one  or  more  null  hy- 
potheses in  only  5  of  the  100  experiments  if  he  uses  an  a    error  rate  of  .05. 

This  error  rate  is  appropriate  when: 

a.  The  experiment  is  the  conceptual  unit. 

b.  The  average  proportion  of  experiments  in  which  one  or  more  faulty  inferences  are  made  is 
to  be  kept  constant. 

c.  The  experimental  error  variance  fluctuates  from  experiment  to  experiment   (see  section  1.5). 

d.  The  value  of  other  inferences  from  the  experiment  is  lowered  as  soon  as  one  faulty 
inference  is  made   (see  section  1.5). 

^Federer,  W.  T.  Error  rates  in  experiments .  (Lecture  notes  from  Advanced  Science1  Seminar  in 
Mathematical  Statistics,  Dept.  of  Math,  and  Statis  .  3  Colo.  State  Univ.,  Ft.  Collins,  Colo.,  Aug. 
7,  1964.) 

^  ,fExperimentwise  "  is  the  term  suggested  by  J.  W.  Tukey  in  the  early  1950 's  to  identify  this 
type  of  error  rate . 


1.4     Duncan's  Protection  Level 


D.   B.   Duncan   (1955)   defined  the  protection  level  concept  for  use  in  a  particular  type  of 
multiple  comparison  test   (see  sections  3.5  and  4.5).  This  concept  is  difficult  to  explain  in  terms 
parallel  to  those  used  in  sections   1.1,   1.2,   and  1.3.  For  this  reason,  discussion  of  the  protec- 
tion level  is   limited  to  an  interpretation  of  its  use  in  Duncan's  New  Multiple  Range  Test. 

An  experiment  is  said  to  be  of  "type  d,"  d  =  1,  2,  k  if: 

i)   It  yields  k  experimental  means,  where  k  >_  d. 
ii)  d  of  the  k  true  treatment  means  are  equal  to  u. 
iii)  The  remaining  k-d  true  treatment  means  are  all  different  from  u  and  all  different  from 
each  other. 

In  other  words,  an  experiment  is  of  "type  d"  if  there  is  only  one  cluster  of  associated  true 
treatment  means,   and  this  cluster  is  of  size  d. 

Now  let  cip  =  the  level  of  significance  the  experimenter  would  choose  for  a  test  of  the  differ- 
ence between  any  two  means,   assuming  that  the  remaining  means  were  not  present.     Then  Duncan's 
protection  level  for  an  experiment  of  "type  d"  =  1  -   (1  -  ap)^~^-  =  (no.  of  experiments  of  type  d 
with  one  or  more  erroneous  inferences)   /   (no.  of  experiments  of  type  d) .     For  example,  with  100 
experiments,  each  yielding  20  treatment  means  and  190  paired  comparisons,   the  experimenter  expects 
to  wrongly  reject  one  or  more  null  hypotheses  in  62  of  the  experiments  if  he  uses  an        error  rate 
of   .05  and  if  each  of  the  100  experiments  is  of  type  20.     If  each  of  the  experiments  is  of  type  2, 
the  experimenter  expects  to  wrongly  reject  one  or  more  null  hypotheses  in  5  of  the  experiments  when 
he  uses  an  otp  error  rate  of  .05. 

This  error  rate  is  appropriate  when: 

a.  The  experiment  is   the  conceptual  unit. 

b.  The  average  proportion  of  experiments  of  type  d  in  which  one  or  more  faulty  inferences  are 
made  is   to  be  kept  constant. 

c.  The  experimental  error  variance  fluctuates  from  experiment  to  experiment   (see  section  1.5). 

d.  The  value  of  other  inferences  from  an  experiment  of  type  d  is   lowered  as  soon  as  one  faulty 
inference  is  made,  but  the  amount  that  the  value  is   lowered  is  small  if  d  is   large  (see 
section  1.5). 

1.5     Discussion  of  a  Error  Rates 

By  comparing  conditions  o  and  d  under  sections  1.1,   1.2,   1.3,  and  1.4,  it  seems  reasonable  to 
conclude  that  in  most  forestry  experiments  an  experimentwise  error  rate,  a^,  should  be  used 
(Hartley  1955).     Errors  of  experimentation  usually  d_o_  affect  the  entire  experiment.     This  is  cer- 
tainly true  when  the  experimenter  uses  a  multiple  comparison  procedure  to  help  him  choose  some 
preferred  treatments.     If  even  one  false  inference  is  made,   the  experiment  provides   little  informa- 
tion about  the  true  relationships  among  treatment  means.     In  forestry  research,  experimental  error 
variance  fluctuations  from  experiment  to  experiment  are  common.     For  example,  varying  climatic 
conditions  affect  the  variance,  but  are  impossible  to  control  from  experiment  to  experiment.  For 
these  reasons,  this  paper  is  slanted  toward  the  use  of  experimentwise  error  rates.     The  experimenter 
must  remember  to  consider  all  four  definitions  of  error  rate,  however,  and  choose  the  one  that  is 
most  appropriate  for  his  situation. 

1.6     The  Terms   "Conservative"  and  "Powerful" 

The  power  of  a  test  is  defined  as  the  probability  of  rejecting  the  null  hypothesis  when  it  is 
false.     Power  is  a  function  of        -  u.,-,   the  actual  mean  differences,  of  a2,   the  common  variance, 
and  of  a,   the  level  of  significance.     If  two  statistical  tests  are  available  for  analyzing  a  set  of 
data,   the  experimenter  should  use  the  test  that  guarantees  the  higher  power  for  the  error  rate  he 
has  chosen. 

Suppose  the  experimenter  wishes  to  compare  a  test  based  on  an  error  rate  per  comparison  with 
a  test  based  on  an  experimentwise  error  rate.     It  can  be  shown  that  if  u.Q  =  a^,   the  test  based  on 
an  error  rate  per  comparison  is  more  powerful.     But  this  is  not  a  true  power  comparison  because  the 
error  rates,  o.Q  and  a^,  are  not  the  same  by  definition.     Although  statisticians  have  compared  them, 
the  power  functions  of  tests  based  on  differently  defined  error  rates  should  not  be  directly 
compared.     For  this  reason,  the  term  "conservative"  is  used  in  the  following  discussion.  The 


statement   "Fisher's  test  is  more  conservative  than  Tukey's  test"  indicates  that  Tukey's  test  is 
more  powerful  in  the  sense  that  the  powers  are  compared  at  ag  =  ay . 

A  simple  way  to  decide  which  of  two  tests  is  more  conservative   (more  powerful  if  the  tests 
are  based  on  the  game  error  rate)   is  to  compare  the  lengths  of  the  1  -  a^,   1  -  a  ,   1  -  a  ,  and 
1  -  otp  confidence  intervals  derived  from  the  tests,  where        =  ae  =        =  (Xp .     The  test  producing 
the  wider  confidence  intervals  about  differences  between  pairs  of  means  is  the  more  conservative. 


GENERAL  MULTIPLE  COMPARISON  TEST  PROCEDURE 


Let  all  k  experimental  means  be  based  on  n  observations  and  have  common  unknown  variance  a  2  In. 
The  general  multiple  comparison  procedure  is   to  order  the  experimental  means  from  smallest  to 
largest,  assigning  consecutive  indices  so  that  X-^  <  X2  <   .  ..<  X]<_i<  X\.     Then  a  table   (see  table  1) 
of  differences  of  all  possible  pairs  is  formed.     Each  table  entry  is  compared  with  an  appropriate 
critical  value  K(a,   v)  s/^n,  where  s2  is  an  independent  estimate  of  a2  based  on  v  degrees  of  free- 
dom and  K(a,  v)   is  a  tabular  value.     If  X^  -  Xj  is  greater  than  K(a,  v)  s/rn,   the  null  hypothesis, 


The  difference, 


Hq:   u-£_  =  uj ,  is  rejected  and  the  alternate  hypothesis,  #j:   u£  4  ]ij,   is  accepted, 

X^  -  Xj,   is  then  said  to  be  "significant."    This  is   the  same  testing  procedure  as  for  the  ordinary 


is  then  said  to  be  "significant 
t-test,  but  with  different  tabled  values 


Table  1 — Differences  between  pairs  of  means 


'k-l 


X 


k-i 


xk  ~  xi 
xk-i  ~  x\ 


x2  -  x1 


xk  ~  xz 
Xk-i  ~  X2 


xk  ~  Xk-i 


I  5*. 

;.  E 

\\t\ 


Multiple  comparisons  based  on  an        error  rate  are  sometimes  better  presented  as  simultaneous 
confidence  intervals,  where  the  probability  is  greater  than  or  equal  to  1-a  that  simultaneously  for 
all  differences  between  pairs  of  treatment  means , 


Jl  -  Jj  -  '{(a,  v)  s/Sn  <]!■£-  Vj  ~        +  #(a,  v)  s/Sn. 


Simultaneous  confidence  intervals  may,   in  turn,  be  regarded  as_tests_  of  hypothesis  if  the  null 
hypothesis  HQ:         =  Uj  is  rejected  when  and  only  when  the  interval  X^  -  Xj  ±  K(a,  v)  s / does  not 
contain  zero. 

The  experimenter  should  try  to  define  his  problem  and  choose  an  appropriate  method  of  analysis 
before  he  looks  at  the  data.     If  tests  of  differences  between  all  pairs  of  experimental  means  are 
paramount,  the  multiple  comparison  procedure  is  to  be  applied  directly — not  after  an  F-test.  Some 
experimenters  use  a  multiple  comparison  procedure  only  if  the  F-test  is  significant.     This  is  not 
an  efficient  approach.     The  F-test  does  not  help  solve  the  problem,  but  only  succeeds  in  making  the 
multiple  comparison  test  more  conservative. 

The  standard  procedure  is  to  calculate  the  experimental  means,  form  an  analysis  of  variance 
table,  use  the  mean  square  error  as  an  independent  estimate  of  the  variance,  and  apply  the 
appropriate  multiple  comparison  test. 
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3.     SOME  MULTIPLE  COMPARISON  TESTS 


There  are  two  types  of  multiple  comparison  tests.     With  one  type,   all  differences^^  -  Xj, 
are  compared  with  the  same  critical  value.     With  the  other  type  the  differences,  X^  -  Xj,  are  com- 
pared with  critical  values  that  depend  upon  the  number  i-j .     These  procedures  are  called  fixed 
range  tests  and  multiple  range  tests,  respectively. 

The  five  tests  discussed  in  this  paper  do  not  exhaust  the  literature  on  multiple  comparison 
methods.     These  tests  are,  however,  the  best  known  and/or  the  most  useful  procedures. 


3.1     L.S.D.  or  Multiple  t-test;  aQ  Error  Rate;  Fixed  Range 

This  test  is  conducted  by  letting  K(a,  v)  =  V*2  t(a,  v)  ,  where  t(a,  v)   is  the  upper  100a/2 
p_ercent  point  of  the  t  distribution  based  on  y  degrees  of  freedom  (Federer  1955).     The  differences 
X^  -  X~,  are  compared  with  the  least  significant  difference  (L.S.D.)  ,   ^  t(a,  v)  sl^n  =  K(a,  v)  s/^n, 
as  described  in  the  general  test  procedure. 

The  a  error  associated  with  the  L.S.D.  test  is  error  rate  per  comparison  ac.     If  the  experi- 
mentwise error  rate  is  important,   this  test  can  be  misleading.     The  following  tabulation  shows  that 
if  20  means  are  to  be  compared,  the  L.S.D.  test  based  on  t(.05,  v)   actually  allows  an        error  of 
90  percent: 

k  =  number  of  means  experimentwise  error 

2  .05 

6  .34 
12  .68 
15  .72 
20  .90 

Needless  to  say,  this  is  not  a  satisfactory  procedure  to  use  if  we  wish  to  control  on  experimentwise 
error  rate . 

The  L.S.D.  test  was  the  first  method  suggested  to  solve  the  multiple  comparison  problem,  but 
recently  has  found  limited  acceptance  among  experimenters. 


3.2     Fisher's  Test;  ag  Error  Rate;  Fixed  Range 


si 


Fisher's  test  is  conducted  exactly  as  the  L.S.D.  test,  except  that  it  is  adjusted  to  insure  an 
error  rate  per  experiment,  ae  (Harter  1957).     This  is  accomplished  by  letting  K(a,  v)  =  ^2mt(a/m,  v) 
where  m  is  the  number  of  paired  comparisons  desired  by  the  experimenter.     Usually,  m  =  k(k-l) /2 , 
the  total  number  of  possible  pairs  of  k  experimental  means.     The  general  procedure  is  followed 
using  the  critical  value  /2  t(a/m,  v)  s / -/n  =  K(a,   v)  s/^n.     Fisher's  test  is  the  most  conservative 
procedure  mentioned  in  this  paper.5     In  fact,  if  k  =  6,  n  =  25,   and  aa  =   .067,   the  corresponding 
experimentwise  error  rate  is  ofo  =   .05   (Harter  1957). 

3.3     Tukey's  Test;         Error  Rate;   Fixed  Range 

Many  textbooks   (Federer  1955,  pp.  22-23;   Scheffe  1959,  pp.   434-436)   contain  tables  of  the 
upper  percentage  points,  q(a,  k,  v) ,   of  the  Studentized  range  for  various  values  of  k,  V,   and  a. 
Tukey's  test_consi_sts  of  finding  q(a,  k,  v)   in  tables  of  the  Studentized  range,  and  comparing  the 
differences  X^  -  Xj  with  q(a,  k,  v)  s/^n  =  K(a,   v)  s I       as  in  the  general  procedure.     The  experi- 
mentwise error,   a^,   for  this  method  is  guaranteed  to  be  less  than  or  equal  to  the  a  in  q(.a,  k,  v)  . 
There  are  other  fixed-range  tests  that  control  on  experimentwise  error  rates   (Scheffe  1959),  but 
Tukey's  test  is  the  most  powerful. 

3.4    Newman-Keuls'  Test ;   ay  Error  Rate;  Multiple  Range 

This  test  also  depends  upon  the  Studentized  range  but  is  less  conservative  than  Tukey's  test 
(Hartley  1955).     The  standard  procedure  is  as  follows: 

5Dunn  (1961)  has  shown  exceptions  to  this  statement  in  eevtain  well-defined  situations. 
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a.  X{,  -  Xj  is  said  to  significant  if 

Xi  -  Xj  >  q(a3  i-j+l,  v)  sl^n,  where  q(a,  i-j+l,  v)  is  the  upper  100a  percent  point  of  the 
studentized  range  for  i-j+l  observed  means  and  V  degrees  of  freedom. 

b.  Construct  the  table  of  mean  differences  as  illustrated  in  section  2. 

c.  Begin  testing  in  the  upper  lefthand  cell.     If  X]^  -  X\_  is  not  significant,  state  that  all 
the  means  are  equal  and  conclude  the  test.     If  Xfc  -  X^  is  significant,   test  Xfc  -  X2  and 

xk-±_  ~  XL-  _  _ 

d.  If  X\  -  X2  is  not  significant,  state_that_J2  =  ^3  =  •••  =  %k  and  test  no  further  differences 
in  the  2nd  through  kth  columns.     If  Xfc  -  X2  is  significant,   test  Xfc  -  X3. 

e.  Continue  in  this  manner,   testing  a  difference,  X^  -  Xj,  only  if  significant  differences 
have  been  found  in  all  cells  above  and  left  of  the  cell  containing  X£  -  Xj. 

If  there  is  only  one  "cluster"  of  true  means,   the  experimentwise  error  is  guaranteed 
to  be  less  than  or  equal  to  a.     But  if  there  are  m  clusters  of  true  means,  aw  is  only 
guaranteed  to  be  less  than  or  equal  to  ma.     A  cluster  is  defined  as  two  or  more  equal  true 
means.     For  example,   if  the  true  relation  among  the  means  is  yj  =  U2<y3=  yi+<...<yk_i  =  yfc 
so  that  there  are  k/2  clusters  of  true  means,   the  experimentwise  error  rate  could  be  as 
high  as   (k/2)a,  which  is  certainly  a  limitation  of  this  procedure. 

3.5     Duncan's  New  Multiple  Range  Test;   <Xp  Error  Rate;  Multiple  Range 

This  test  procedure  is  conducted  exactly  as  the  Newman-Keuls'  test  except  that  the  tabled 
values,  q(a,  i-j+l,  v) ,  have  been  revised  to  provide  a  less  conservative  test   (Duncan  1955).  To 
use  this  method,   the  revised  values,  q*(a,  i-j+l,  v) ,  must  be  read  from  special  tables,  usually 
titled  "Critical  Values  for  Duncan's  New  Multiple  Range  Test"  (Federer  1955;  Harter  1960b).  These 
values  are  slightly  smaller  than  corresponding  values  in  a  standard  Studentized  range  table. 
Duncan  justifies  this  method  by  defining  the  protection  level  concept  and  argues  that  the  experi- 
mentwise error  rate  is  too  strong  a  criterion.     To  illustrate  the  difference,  if  k  =  6  and  n  =  25, 
Duncan's  test  must  be  run  with  tabled  values  of  q*(.01,  i-j+l,  v)   to  insure  an  experimentwise  error 
of  .05   (Harter  1957)  . 

Duncan's  test  is  affected  in  a  more  complicated  manner  than  the  Newman-Keuls1   test  if  there  are 
m>l  clusters  of  the  true  means.     This  fact  does  not  seem  to  be  mentioned  in  the  literature. 

3.6     Selecting  an  Appropriate  Test 

In  the  preceding  discussion,   the  term  conservative  was  freely  used.     Power  is  a  more  meaningful 
concept  to  consider  when  selecting  the  better  of  two  tests.     To  find  a  true  comparison  of  the  powers 
of  these  five  multiple  comparison  tests,  each  method  can  be  adjusted  to  yield  the  same  experiment- 
wise  error  rate.     If  this  is  done,  computations  indicate  that  all  five  tests  have  roughly  the  same 
power . 

Since  the  powers  are  much  the  same,   the  choice  of  a  test  is  actually  governed  by  the  choice  of 
an  appropriate  a  error  rate.     By  considering  the  definitions  and  discussion  of  error  rates  given  in 
section  1,  the  experimenter  can  avoid  the  confusion  previously  associated  with  the  selection  of  a 
multiple  comparison  test.     Note  that  if  k  =  2,  all  five  methods  are  identical  in  both  error  rate 
and  critical  value. 

Because  the  experimental  error  variance  fluctuates  from  experiment  to  experiment  in  forestry 
work,  the  o.q  and  ag  error  rates  are  not  adequate  in  most  cases.     This  fact  generally  eliminates  the 
L.S.D.  test  and  Fisher's  test  from  consideration. 

If  it  is  assumed  that  the  experimenter  believes  the  true  treatment  means  are  not  grouped  in 
more  than  one  cluster,  how  does  he  choose  between  Tukey's  test,  Newman-Keuls'  test,  and  Duncan's 
test?     An  answer  to  this  question  depends  upon  the  individual  philosophy  of  the  experimenter. 

If  he  feels  that  the  importance  of  a  false  inference  diminishes  as  the  number  of  true  treatment 
means  in  the  cluster  increases,  he  will  use  the  Duncan  procedure.     In  using  Duncan's  test,  he  real- 
izes that  the  probability  of  falsely  rejecting  one  or  more  hypotheses  is  somewhere  between  a-g  and 
l-(l-ap)^-1.     For  the  example  in  section  1.4  where  k  =  20,  a^  =   .05,  and  l-(l-ap)^-1  =  .62. 
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If  the  experimenter  decides  that  one  false  inference  decreases  the  value  of  the  other  infer- 
ences, regardless  of  the  number  of  means  in  the  cluster,  he  wants  to  control  on  an        error  rate. 
This  is  usually  the  case  in  forestry  research.     Now  either  Newman-Keuls T   test  or  Tukey's  test  can 
be  chosen.     The  experimenter  should  realize  that  if  he  believes  the  Newman-Keuls'  test  at  ciy  =  .05 
is  too  conservative,  he  can  use  a  larger  oty  error  rather  than  adopting  Duncan's  procedure. 

It  is  true  that  the  experimenter  can  not  know  the  relationship  among  the  true  treatment  means . 
His  guess  as  to  the  number  of  clusters  of  true  treatment  means  is  based  on  his  experience  and  know- 
ledge of  the  experimental  situation.     The  statisticians  can  only  state  that  if  experimentwise  error 
rate  is  important  and  the  experimenter  feels  that  the  true  means  are  not  grouped  in  more  than  one 
cluster,   the  Newman-Keuls'   test  is  appropriate.     If  the  experimenter  suspects  that  the  true  means 
are  grouped  in  two  or  more  clusters,  he  should  use  Tukey's  test. 

An  experimenter  can  provide  appropriate  answers  a  large  percent  of  the  time  if  he  routinely 
applies  Tukey's  test  to  every  multiple  comparison  problem  he  encounters.     This  is  true  because  gen- 
erally experimentwise  error  is  important,   and  because  Tukey's  test  is  valid  no  matter  how  the  true 
means  are  grouped. 

Sometimes  the  experimenter  wishes  to  compare  all  treatments  with  a  standard  or  a  control.  He 
may  feel  that  only  differences  between  treatment  means  and  the  control  mean  are  of  interest.  The 
Newman-Keuls'   test  is  appropriate  for  this  situation.     Dunnett   (1955)  proposes  a  fixed-range  test 
for  this  case  that  requires  a  special  set  of  tables  . 


3.7     Means  Based  on  Unequal  Numbers  of  Observations 

One  of  the  assumptions  made  in  section  2  was  that  the  experimental  means  are  all  based  on  the 
same  number  of  observations.     If  this  is  not  true,    (i.e.         is  based  on        observations  and  has  a 
variance  of  o2/n^)   the  tests  must  be  adjusted  (Sarhan  and  Greenberg  1962)  . 

To  adjust  the  L.S.D.  test  and  the  Fisher 's_  test,  multiply  the  usual  tabled  value  by 
s(l/n-i  +  1/n-y)  1//2  to  test  the  difference  X{_  -  Xj.     The  errors,  ac.  and  ae ,  are  not  affected. 

To  adjust  Tukey's  test,  Newman-Keuls'   tes_t ,  and  Duncan's  test,  multiply  the  usual  tabled  values 
by  i/s2/2(l/n^  4-  to  test  the  difference  X^  -  Xj.     This  is  a  conservative  approximation  for  all 

three  methods   (Kramer  1956)  . 


4 .  EXAMPLE 

Hartley  (1955)  quotes  data  on  the  comparative  yields  of  naphthalene  dye  stuff  prepared  from 
k  =  6  different  "treatments"  of  H-acid.     Each  treatment  was  replicated  n  =  5  times.     The  means, 
arranged  in  ascending  order,  are  given  below. 

Treatment  index, _i               12              3  4             5  6 

Treatment  mean,  X^        1,470      1,498      1,505  1,528      1,564  1,600 

The  analysis  of  variance  shown  below  provides  an  estimate  of  the  variance,  s2  =  2,451,  with  v  =  24 
degrees  of  freedom: 

Degrees  Sum  of  Mean 

of  freedom  squares  square 

Due  to: 

Between  treatments                 5  56,360  11,272 

Within  treatments               _2A  58,824  2,451 

Total                           29  115,184 

Next  a  table  of  mean  differences  is  formed  (table  2) . 
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Table  2 — Differences  between  experimental  means 


X2  X3  X\  X5  X6 


Xc 


X, 


X. 


X, 


130  102 


94 


58 


35 


28 


66 


30 


95 


59 


23 


72 


36 


26 


4.1     L.S.D.  Method  -  aG  =  .05 
£(.05,  24)  =  2.06;     /2  t(a,  v)  s / =  /2(2.06)    (22.14)  =  64.6 

The  L.S.D.,  64.6,  is  compared  with  the  values  in  table  2  and  the  results  are  conveniently  summarized 

by    Xl  X2  X3  Xh  X5  X6. 


Any  two  means  not  underscored  by  the  same  line  are  significantly  different. 

Any  two  means  underscored  by  the  same  line  are  not  significantly  different,   as  defined  in  section  2 

4.2     Fisher's  Test  -  ae  =  .05 
£(.05/15,  24)  =  £(.0033,  24)  =  3.429,       /2  t(a,  v)  s I  Jn  =  /2(3.429)   (22.14)  =  107.0 
The  critical  value,  107_J3,  is  compared  with  the  mean  differences  in  table  2,  and  the  results  are 


summarized  as  in  4.1 


X  -I  X r.  X  ^  X  ,  X  r-  Xr 

1  Z  O  H  0  D 


4.3    Tukey's  Test  -  ct^  =  .05 

From  tables  of  the  Studentized  range,     (7(01,  k,  v)  =  q(.05,  6,  24)  =  4.37. 

K(a,  v)  s/Jn=  (4.37)    (22.14)  =  96.8 

The  critical  value,  96.8_j_  is_comp_ared_  with  the  mean  differences  in  table  2,  and  the  results  are 


summarized  as  in  4.1 


x~    x ~    X ~    X.     X r    X r 

1        2.        6        4        b  b 


4.4    Newman-Keuls  '  Test  -  ot^  =  .05 

To  find  the  critical  value  for  the  Y&  -  7j  cell,  find  q(.05,  6-1+1,  24)  =  4.37  in  the  Studentized 
range  table  and  multiply  by  s / Sn  =  22.14,  so  that  4.37(22.14)  =  96.8. 

To  find  the  critical  value  for  the  X5  -  X2  cell,  find  ^(.05,  5-2+1,  24)  =  3.90  in  tables  of  the 
Studentized  range  and  multiply  by  s/Sn  ,  so  that  3.90(22.14)  =  86.3. 

The  test  was  concluded  when  ^6  z.  Xi*>-  X2^>  aE_^  Xh.  ~  Xl  were  aH  found  nonsignificant.  The 

results  are  summarized  as  in  4.1.      X\    X2    X3    Xh-    X5  Xs 
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Table  3 — Critical  Values  for  Newman-Keuls '  Method  with  a  =  .05, 

K  =  6,  n  =  24 


x1        x2        x3        xh        x5  x& 

Z6  96.8*      92.3*      86.3*  78.2 

X5  92.3*  86.3 

Xh  86.3 

X2 

*Indicates  significance  for  data  in  table  2. 

4.5  Duncan's  New  Multiple  Range  Test  -  ctp  =  .05 

To  find  the  critical  value  for  the  T&  -  X    cell,   find  q*(.05,  6-1+1,  24)  =  3.28  in  a  table  of 
critical  values  for  Duncan's  new  multiple  range  test  and  multiply  by  s/ffl,  so  that  3.28(22.14)  =  72.6. 

To  find  the  critical_value  for  the  X^  -  X1  cell,  find  q*(105,   4-1+1,   24)  =  3.16  in  the  same 
table  and  multiply  by  s / vn  so  that       3.16(22.14)  =  70.0. 

The  test  was  concluded  when  Zg  -  X$ ,  X§_  -  X2,  and  Zi+  -_Xl  ,_were  all  found  nonsignificant.  The 
results  are  again  summarized  as  in  4.1.        X\    X2    X3    X^    Z5  ^6 


Table  4  — Critical  Values  for  Duncan's  Method  with  a  =  .05, 

K  =  6}  v  =  24 


x1  x2 


X  „  X.  X  r-  X 

3  h  5  6 


X, 


X, 


72.6*      71.5*      70.0*      68.0*  64.6 


71.5*  70.0 


70  .0 


*Indicates  significant  cells  for  data  in  table  2 
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5.     TESTING  MULTIPLE  CONTRASTS  AMONG  TREATMENT  MEANS 


Many  times  an  experimenter  is  interested  in  "multiple"  contrasts,  which  are  differences 
between  groups  of  means.     The  data  may  suggest  relationships  among  treatment  means   that  previously 
were  not  apparent,  or  the  experimenter  may  wish  to  test  some  multiple  contrasts  for  which  independ- 
ent sums  of  squares  cannot  easily  be  partitioned  in  the  analysis  of  variance  table. 

A  linear  contrast  among  k  true  treatment  means  is  defined  as 

k 

=  I  Cm\im  =  Ciui+C2u2+.  •  -+CkVk> 

m=l 

where  the  Cm's  are  any  constants  subject  to  the  condition: 

i 

K 

I  fin    =  Ci  +  C2  +...+  Ck  =0. 

m=l 

One  possible  multiple  linear  contrast  among  six  means  is 

\p  =       +  v5)/2  -  (y2  +  ^4  +  ve)/3, 

where  the  coefficients,  Cl  =  1/2,  C2  =  -1/3,  C3  =  0,  C\  =  -1/3,  C5  =  1/2,  C6  =  -1/3,  sum  to  zero. 

Differences  between  pairs  of  treatment  means  are  actually  simplified  forms  of  multiple  contrasts 
where  each  group  of  means  contains  only  one  member.     In  notation, 

k  1,  if  m  =  £  k 

ui  -  yj  =  I  CmVm,  where  Cm  =  -1,  if  m  =  J  and     \  Cm  =  l+(-l)  =  0 

m=l  0,  otherwise  m=l 

Methods  for  simultaneously  testing  such  "simple"  linear  contrasts  are  described  in  sections  2,  3, 
and  4 . 

There  are  two  general  methods  of  placing  simultaneous  confidence  intervals  around  multiple 

contrasts  among  treatment  means   (Scheffe  1959).     Both  provide  an  experimentwise  error  rate. 

5.1     S-Method   (Scheffe)  -  cty  Error  Rate 

Under  the  usual  assumptions   (X\ ,  Xfc  are  independently,  identically,  normally  distributed 

random  variables),   the  probability  is  1-a  that  simultaneously  for  all  possible  independent  linear 
contrasts , 

k             k     _          k  k  k 

$  =    I  Cm\im,     \  CmXm-  S{  l/2s/^  <  ^  <    j;  CmXm+  S(  lcm2)  l/2s/Jns 

m=l          m=l              m=l  rn=l  m=l 

where  . 

S  =  {(k-l)Fa,  Vxl/2, 

s  is  the  estimate  of  a  based  or.  V  degrees  of  freedom,   and  each  experimental  mean  is  based  upon  n 
observations . 

5.2     T-Method   (Tukey)  -        Error  Rate 

Under  the  same  assumptions  as  in  5.1,  the  probability  is  1-a  that  simultaneously  for  all 
contrasts , 

k  k  k  k  k 

*  =    I  cmVm*  I  Vm-  q(**  K  v)s/Sn  -1/2  J     \Cm\<  4>  <  I  Vm+q{a,  k3  v)s/Sn-l/2  I  \Cm\ , 

m=l  777=1  777=  1  777=1  777=1 

where  q(a,  k,  v)  is  the  upper  100a  percent  point  of  the  Studentized  range  based  on  k  means  and  V 
degrees  of  freedom,  s  and  n  are  as  in  5.1,   and   \C \   is  the  absolute  value  of  Cm. 

Notice  that,  if  the  contrast  is  a  simple  linear  contrast, 

k 

1/2  7  \C  I  =  1/2  {111  +  |-l|}  =  1 


777=1 


777' 


so  that  the  interval  above  is  precisely  that  given  by  Tukey ' s  method  of  section  3.3  . 
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5 . 3    Dis  cussion 


These  confidence  intervals  may  be  used  to  provide  a  test  of  hypothesis.     Reject  the  null 
hypothesis , 

H0:i>  =  J  CmVm  =  0, 

if  the  confidence  interval  does  not  contain  0,   and  accept  the  null  hypothesis  if  the  confidence 
interval  does  contain  0. 

As  pointed  out  in  section  3.3,   the  T-method  is  more  powerful  for  testing  simple  linear  con- 
trasts.    It  has  been  shown  that  the  5-method  is  more  powerful  for  testing  multiple  linear  contrasts 
(Scheffe  1959).     Thus  a  good  suggestion  is  to  use  the  T-method  if  only  simple  linear  contrasts  are 
to  be  tested  or  if  a  mixture  of  simple  and  multiple  linear  contrasts  are  to  be  tested.     Use  the 
iS-method  if  only  independent  multiple  linear  contrasts  are  to  be  tested. 

The  S-method  is  valid  for  testing  either  simple  or  multiple  contrasts  among  adjusted  means 
from  an  analysis  of  covariance   (Halperin  and  Greenhouse  1958) . 

Sometimes  the  assumption  of  independence  among  treatment  means  is  not  reasonable.     But  if  the 
experimenter  believes  that  all  the  covariances  between  means  are  equal,   an  extension  of  the  T-method 
may  be  used  to  test  linear  contrasts  among  the  means   (Scheffe  1959)  . 

5.4  Example 

Consider  the  data  presented  in  section  4.  Suppose  treatments  3,  5,   6  were  administered  in  the 

presence  of  an  impurity  and  treatments  1,  2,  4  were  free  of  the  impurity.     The  experimenter  may 

feel  that  it  is  important  to  test  the  multiple  contrast 

\p  =  \i1  +  y2  -  y3  +  \ik  -  u5  -  w6. 

5-method,  a    =  .05 
w 

k 

I  CmXm  =  1470  +  1498  -  1505  +  1528  -  1564  -  1600 

=  -173 

k 

I  Cmz  =  l2  +  l2  +  (-1)2  +  l2  +  (-1)2  +  (-1)2  =  6 
m=i 

F(.05,  5,  24)  =  2.62 
s/Jn  =  22.14 

S  =  {5(2.62)  W2  =  3.62 

Therefore , 

Pr  {-173  -  (3.62)  (6)  1/2(22.14)   <  ijj  <  -173  +  (3  .62)  (6)  1/2  (22 . 14)  }  =  .95 

or 

Pr  {-369.4  <  ii  <  23.4}  =  .95 

T-Method,        =  .05 

Z     CfljXjn  =  -173 
m=l  k 

l\Cm\  =  1  +  1  +  J  —X I  +  1  +  |-l|  +  |-l|  =6 
m=l 

q(.05,  6,  24)  =  4.37 
s/Sn  =  22.14 

Therefore , 

Pr  {-173  -  (4.37)  (22.14)  (6/2)  <  i\j  <  -173  +  (4  .37)  (22 . 14)  (6/2)  }  =  .95 

or 

Pr  {-463.3  <  ip  <  117.3}  =.95 


The  results  of  both  methods  suggest  that  the  average  of  means  of  treatments  containing  the 
impurity  were  not  significantly  different  from  the  average  of  means  of  treatments  not  containing 
the  impurity. 

This  example  shows  that  confidence  intervals  about  a  multiple  linear  contrast  can  be  much 
wider  with  the  T-method  than  with  the  S-method. 
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